…old (lance-format#7563)

## Problem

`StoredBatch::estimate_batch_size` (mem_wal `batch_store.rs`) summed
`Array::get_array_memory_size`, which counts every buffer's full
`capacity()` **regardless of the array's offset/length**. Arrow's own
docs are explicit: a sliced `ArrayData` "may only refer to a subset of
the data ... but the size returned includes the entire size of the
buffers," and slices sharing a buffer "will both report the same size."

When ingest slices one incoming batch into N zero-copy WAL chunks, each
slice therefore reports the **whole shared parent buffer**. This
estimate feeds `maybe_trigger_memtable_flush` (`estimated_size() >=
max_memtable_size`), so the active memtable flushed far below the
configured size — e.g. a 250 MB target tripping at a few MB of real
data.

## Fix

Use Arrow's slice-aware `ArrayData::get_slice_memory_size`, which
reports only the slice's own window, with a fallback to the old call for
the rare types where it errors (conservative — over-counts, never
under).

## Evidence

A standalone repro (100 zero-copy slices of a 20 MB parent, the common
ingest pattern):

| | estimate |
|---|---|
| actual retained heap | ~20 MB |
| old (`get_array_memory_size`) | **~2744 MB** (131× over-count) |
| new (`get_slice_memory_size`) | ~20 MB |

Even a single owned (non-sliced) batch over-counts ~1.26× under the old
path, so the new estimate is strictly more accurate.

## Test

Adds `test_estimated_size_is_slice_aware`: tiles a parent into 100
zero-copy slices, appends them, and asserts the store's running estimate
covers the real payload without the ~N× blow-up (guarded against the old
over-counting sum).

`cargo test -p lance --lib batch_store`, `cargo fmt --all`, and `cargo
clippy -p lance --tests -- -D warnings` all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e-format#7644)

`crossbeam-epoch` 0.9.18 has an invalid pointer dereference in the
`fmt::Pointer` impl for `Atomic`/`Shared` when the underlying pointer is
invalid
([RUSTSEC-2026-0204](https://rustsec.org/advisories/RUSTSEC-2026-0204)).
It is pulled in transitively via `rayon` → `crossbeam-deque`, and is now
failing the `cargo-deny` CI check on all PRs.

This bumps `crossbeam-epoch` to the fixed version (>=0.9.20) across all
three lockfiles (`Cargo.lock`, `python/Cargo.lock`,
`java/lance-jni/Cargo.lock`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ance-format#7469)

## Problem
Creating a single fragment via `LanceFragment.create` /
`FragmentCreateBuilder` with an Arrow JSON column (`arrow.json`, stored
as Utf8) writes raw UTF-8 bytes into a column whose schema declares
Lance JSON (JSONB / LargeBinary), corrupting subsequent reads.

## Root cause
The multi-fragment and dataset write paths run the Arrow JSON -> Lance
JSON conversion via `do_write_fragments`. The single-fragment create
path skipped it.

## Fix
Run the same conversion in the create path via
`SchemaAdapter::to_physical_stream`.

## Test
`test_fragment_create_with_json_column` (Python).

Co-authored-by: xiaojiebao <xiaojiebao@xiaomi.com>

…ns (lance-format#7634)

Follow-up to the review discussion on lance-format#7601: fuzzy expansion previously
ran
inside every partition with a per-partition `max_expansions` cap, and
the
per-partition picks were unioned afterwards. The effective cap was
`num_partitions × max_expansions`, so the same query could match more
terms —
and return different results — purely because of how the corpus happened
to be
partitioned (e.g. after a tail-heavy build splits its leftovers).

## What

- **Expansion now runs once per query**, in
`InvertedIndex::bm25_search`,
  under the query-wide `max_expansions` budget. For each query token the
per-partition FST candidates merge into one lexicographically ordered
set
and the remaining budget takes a prefix of it. Each partition is only
asked
for its `remaining` lexicographically smallest candidates, which is
lossless
for that selection: any term among the merged lex-smallest `remaining`
is
  also among its own partition's smallest `remaining`.
- **Partitions receive the final token list** and no longer expand
(`load_posting_lists` drops its `expand_fuzzy` call); `params.fuzziness`
  still drives the grouped fuzzy dedup/scoring semantics.
- **The BM25 scorer reuses the same expansion**
(`bm25_scorer_for_final_tokens`),
so scorer terms and searched terms stay in lockstep and a query pays for
one
  expansion instead of one for the scorer plus one per partition.
- **Fuzzy AND/phrase keep their contract** — every original token
position
must retain at least one expansion — via an index-level check
(previously
  implicit in each partition's own expansion).
- `InvertedPartition::expand_fuzzy` stays for API compatibility,
reimplemented on the shared per-token candidate collector;
single-partition
behavior is unchanged (same FST lexicographic order, same budget fill).

## Semantics

Single-partition indexes behave exactly as before. Multi-partition
indexes
previously over-expanded in proportion to their partition count; they
now
honor the documented cap, and the expansion (hence the result set) is a
pure
function of the segment's vocabulary, independent of partition shape.

## Tests

- `test_fuzzy_expansion_cap_is_global_across_partitions`: two partitions
with
disjoint variants, binding cap → exactly the 3 lexicographically
smallest
  terms across both (fails on main, which returns 4).
- `test_fuzzy_results_independent_of_partition_shape`: the same four
docs
built as one partition and as two return identical `(row_id, score)`
sets
  under a binding cap (fails on main).
- Existing fuzzy suite (grouped AND, position grouping, whole-query cap)
unchanged and passing. The non-binding-cap regression test over the real
  tail-split builder path lives in lance-format#7601.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Yang Cen <yang@lancedb.com>

## Summary

Close lance-format#5887

Build FTS indexes for list string columns as row-level documents instead
of flattening each list element into its own document.

This means `List<Utf8>`, `List<LargeUtf8>`, and their `LargeList`
variants now treat a row as one document, with non-null list elements
contributing text fragments to that document. Token positions are
continuous across list elements, so phrase queries can match across
element boundaries.

The change intentionally does not add user parameters, persistent
metadata, legacy migration, or query-side deduplication. Old list
indexes keep their existing behavior.

## Context

Issue lance-format#5887 reports duplicate FTS results for list string columns
because the old index builder treated each list element as a separate
document with the same row id. Row-level indexing fixes the result
duplication and brings BM25 document statistics back to row-level
semantics for newly built indexes.

Follow-up issue lance-format#7654 tracks MemWAL in-memory FTS support for list
string columns.

## Validation

- `cargo fmt --all`
- `git diff --check`
- `cargo test -p lance-index flat_bm25_search`
- `cargo test -p lance-index test_worker_`
- `cargo test -p lance test_fts_list`
- `cargo test -p lance test_fts_index_with_`
- `cargo clippy -p lance-index --tests -- -D warnings`
- `cargo clippy -p lance --tests -- -D warnings`

## Why

This is the diagnostics layer for the large S3/cloud scan work. The
final goal is to move large scans close to the raw scheduler / storage
bandwidth ceiling while keeping point `take` and small reads from
regressing.

Before changing scan policy, scheduler capacity, page loading, or decode
behavior, we need reproducible visibility into where time and
backpressure are spent across the raw scheduler, direct file reader,
scanner, and dataset take paths.

## What

- Add scheduler diagnostics snapshots for standard and lite schedulers.
- Expose queue state such as active/pending IOPS, pending bytes, byte
budget, and head-of-queue blocking state.
- Add a hidden scanner diagnostics callback for benchmark tooling.
- Add `s3_file_reader_diagnostics`, a harness-free JSONL diagnostics
benchmark for:
  - raw scheduler reads
  - direct `FileReader` scans
  - full `Scanner` scans
  - `Dataset::take`

## Validation

- `cargo test -p lance-io
test_standard_scheduler_diagnostics_tracks_queue_state -- --nocapture`
- `cargo test -p lance
test_scan_scheduler_diagnostics_callback_is_called -- --nocapture`
- `cargo check -p lance --bench s3_file_reader_diagnostics`
- `git diff --check`

S3 smoke validation confirmed that the diagnostics output can report raw
scheduler and scanner counters, including scheduler bytes, active IOPS,
pending IOPS, and throughput samples.

## Boundaries

This PR is instrumentation only.

It does not change default scan scheduling, scheduler capacity,
unordered scan semantics, structural page loading, fullzip decoding,
storage format, wire format, or public data behavior. Throughput
improvements are expected in later PRs that use these diagnostics to
validate bottleneck-specific changes.

…ance-format#6555)

In `dataset.py` line 6584-6587, the condition to check for numpy arrays
was:

```python
elif isinstance(query, (list, tuple)) or (
    _check_for_numpy(query),
    isinstance(query, np.ndarray),
):
```
The expression (_check_for_numpy(query), isinstance(query, np.ndarray))
creates a tuple (bool, bool), which is always truthy (a non-empty
tuple). This means any input that is not a list, tuple, pa.Scalar, or
pa.Array would incorrectly enter the numpy conversion branch. This will
result in an unexpected conversion error: ValueError: could not convert
string to float: np.str_('not a vector')

…e-format#7643)

`spawn_cpu` runs work on a dedicated pool sized to
`get_num_compute_intensive_cpus()`,
which collapses to a **single blocking thread** on hosts with `<= 3`
CPUs. A closure
that ever *waits* — blocking channel send/recv, I/O, a contended lock,
or `block_on` —
parks that thread and can starve the very work that would unblock it,
deadlocking the
pool with a silent 0% hang. This is the failure fixed in lance-format#7423.

This PR writes the rule down and audits the call sites:

- Expand the `spawn_cpu` doc comment with the "must never wait on
anything" rule
(no channels / no I/O / no locks / no `block_on`), the rationale, and a
pointer to
the recommended pattern (keep the waiting in async code, hand only pure
CPU work to
  `spawn_cpu`).
- Add a concise Concurrency rule to `rust/AGENTS.md`.

### Audit

All ~20 `spawn_cpu` call sites were reviewed, following transitive
calls. Every
production site is clean (I/O/loading is awaited before the closure; the
closures do
pure in-memory CPU work) except one:

- **IVF streaming partition search** (`ivf/v2.rs`): a single `spawn_cpu`
closure does
`blocking_recv` + `blocking_send` on capacity-1 channels. This is a
deliberate
optimization from lance-format#6475 (run a query's whole sequential search on one
CPU worker to
avoid per-partition fan-out, a measured 14–30% latency win), so fixing
it trades a
benchmarked perf win against small-host correctness and needs the lance-format#6475
author's
  input. Tracked in lance-format#7642 rather than fixed here.

The FTS builder site is already correct after lance-format#7423.

Closes lance-format#7637

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…format#6581)

## Summary

- Fix panic `InvalidArgumentError("Max offset of X exceeds length of
values Y")` in `ListArray::new` during `to_table(filter=...,
columns=[list_struct_col, ...])` on v2.0 datasets.
- Root cause: `merge_with_schema` (called from `TakeStream::map_batch`)
passed `left_list.trimmed_values()` alongside
`left_list.offsets().clone()`. When the left list is a sliced view (e.g.
a filtered batch), offsets do not start at zero and reference positions
past the end of the trimmed child, panicking in `ListArray::new`.
- Add `ListArrayExt::trimmed_offsets()` that returns offsets shifted to
start at zero, and use it in the `List`/`LargeList` branches of
`merge_with_schema`.

## Test plan

- [x] New regression test `test_merge_with_schema_sliced_list_struct` in
`lance-arrow`: fails on `main` with the exact panic, passes with the
fix.
- [x] All existing `lance-arrow` merge tests still pass (9/9).
- [x] Python repro from the issue (1M rows, `200k + 800k + 14650`
sparse-tail pattern) no longer panics and returns the expected 214,650
rows with correct data (verified against a manually-filtered reference
batch).

Fixes lance-format#6580

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

## Summary
- Update the MemWAL table-format spec to match the current inline index,
WAL, shard manifest, flushed generation layout, and reader semantics.
- Document forward flushed row ordering, deletion-vector primary-key
deduplication, sidecars, and implemented sharding transforms.

Validation: `git diff --check`; `cd docs && uv run mkdocs build`.

…ormat#7533)

Adds an optional `metrics` feature to `lance-io` that publishes object
store metrics through the [`metrics`](https://docs.rs/metrics) crate
facade, so applications can wire them to Prometheus, OpenTelemetry, etc.
without Lance depending on a specific backend. The feature is **off by
default**.

Two layers cooperate:

- **`MeteredObjectStore`** wraps any `ObjectStore` and records
per-operation request counts, transferred bytes, latency, errors, and
in-flight count, labelled by `operation`
(get/put/head/list/delete/copy/rename) and `scheme`. Works for every
store regardless of backend.
  - `lance_object_store_requests_total{operation, scheme}`
  - `lance_object_store_request_bytes_total{operation, scheme}`
- `lance_object_store_request_duration_seconds{operation, scheme}`
(histogram)
  - `lance_object_store_errors_total{operation, scheme}`
- `lance_object_store_in_flight_requests{operation, scheme}` (gauge) —
requests currently outstanding, tracked by an RAII guard so the count
stays balanced even if a request future or a list/delete stream is
dropped before finishing.
- **`MeteringHttpConnector`** wraps the HTTP client used by the native
cloud stores (S3 / GCS / Azure) via `with_http_connector`, and records
throttle responses per attempt:
- `lance_object_store_throttle_total{status, scheme}` — counts 429 / 5xx
responses.

Because object_store's retry loop re-issues each request through the
`HttpService`, this observes **every retried 429/503 with its precise
status code** — something a store-level wrapper cannot see, since
object_store retries internally below the `ObjectStore` trait.

Multipart part uploads (`put_part`) record the same count / bytes /
latency / errors / in-flight set as a unary `put`, so multipart writes
are consistent with the rest of the object store metrics.

### Success criteria
- [x] Requests measured with counts, bytes, latency, and error count,
across `operation` and `scheme` labels.
- [x] Retry/throttle counts separated by reason (429 vs 503) — stretch
goal.
- [x] `metrics` is an optional dependency.

### Notes
- Opendal-backed stores (tos/oss/goosefs/tencent/hf) get the store-level
metrics but not the HTTP-level throttle metrics, since they bypass
object_store's HTTP client.
- Enable with `lance-io`'s (or `lance`'s) `metrics` feature.

Closes lance-format#7504

🤖 Generated with [Claude Code](https://claude.com/claude-code)

### Documentation
Available metrics are catalogued in one shared table
(`rust/lance/src/metrics.md`), surfaced both in the Rust
`lance::metrics` module docs and a new **Observability** docs-site page
(`docs/src/guide/observability.md`) via an mkdocs snippet include —
single source of truth for Rust and Python.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ormat#7589)

## What

The FSST decoder needs its output buffer to be at least `8x` the
compressed input, but the size guard only required `3x` and the public
`decompress` doc claimed "at least 3 times". This raises the guard to
`8x` (with 32-bit overflow safety), fixes the docs, documents the
`unsafe` invariants, and aligns the in-tree example.

## Why

`decompress_bulk` writes a full 8-byte word per code but advances the
output cursor by only the symbol length, relying on a later write (or
spare capacity) to overwrite the slack — the standard FSST decode trick:

```rust
// fsst.rs, final symbol of a run — no following write to cover the slack
let code = compressed_strs[in_curr] as usize;
unsafe {
    ptr::write_unaligned(out.as_mut_ptr().add(*out_curr) as *mut u64, src); // writes 8 bytes
}
*out_curr += lens[code] as usize; // advances by len (1..=8)
```

`MAX_SYMBOL_LENGTH` is 8, so a 1-byte code expands to at most 8 bytes
and the whole decoded output is at most `8x` the input. The last code of
a run is the tightest case: it has no following write, so its 8-byte
store must still land inside the buffer. Both facts require the buffer
to be at least `8x`.

The guard in `FsstDecoder::init` was:

```rust
// when decoder_switch_on is true, we make sure the out_buf is at least 3 times the size of the in_buf,
if self.decoder_switch_on && in_buf.len() * 3 > out_buf.len() {
    return Err(...);
}
```

`3x` is too weak: it accepts a buffer between `3x` and `8x` that the
decode loop then writes out of bounds. This is **not reachable from the
in-tree decompressors** (`FsstPerValueDecompressor`,
`FsstMiniBlockDecompressor`, and the v2.0 `FsstPageDecoder`) — they
already allocate `in_buf.len() * 8`, so the `3x` check never fires for
them. But the crate's own `benchmark` example allocated `3x` and would
now be rejected, and the guard was unsound for any future caller that
trusted the "3 times" wording.

## Changes

- Raise the guard from `* 3` to `* 8`, using `checked_mul(8)` so a `>=
512 MiB` input can't wrap `len * 8` on a 32-bit target and bypass the
check (overflow is treated as "too small"). This is a threshold
comparison, not an allocation — no extra memory: existing `8x` callers
still pass, and a `<8x` buffer is now rejected instead of silently
overflowing.
- Fix the misleading "3 times" wording in the guard comment and the
`decompress` pub-doc to `8x`, with the 8-byte-write reason.
- Add `// SAFETY:` comments to the unaligned load/store blocks in
`fsst_unaligned_load_unchecked` and `decompress_bulk`, per the repo
convention of documenting every `unsafe` block. The comments honestly
disclose which conditions are *proven* (loop guards, the `8x`
output-buffer check, the symbol-table size check) versus which are
*trusted preconditions* for well-formed input (`lens[code] <= 8` and
offset values are loaded verbatim from the symbol table / offsets and
are not re-validated on decode). Hardening the decoder against corrupt
on-disk structures is a separate concern, out of scope here.
- Update the `benchmark` example's decode buffer from `3x` to `8x` so it
satisfies the corrected guard.
- Add a regression test
`test_decompress_rejects_undersized_output_buffer` asserting a
1-byte-short buffer is rejected and an exact-`8x` buffer succeeds (fails
against the old `3x` guard).

## Test plan

- [x] `cargo test -p fsst` — 4/4 pass (incl. new test).
- [x] `cargo test -p lance-encoding fsst` — 12/12 pass (all consumers).
- [x] `cargo fmt -p fsst --check` — clean.
- [x] `cargo clippy -p fsst --tests --examples -- -D warnings` — clean.

…m_vec (lance-format#7614)

## What

Follow-up to lance-format#7500 (wjones127's suggestion in
lance-format#7500 (comment)).
`BFloat16Array::from(Vec<bf16>)` still walked the input a second time,
copying two bytes per element into a freshly allocated `MutableBuffer`.
This hands the input `Vec` straight to `Buffer::from_vec`, so the
existing allocation is reused as-is — no per-element copy, no
`MutableBuffer::extend` loop.

`Buffer::from_vec` requires `T: ArrowNativeType`, which `bf16` does not
implement (arrow-rs registers the trait only for `f16`/`f32`/`f64` among
the float family; Apache Arrow has no bf16 primitive type, which is why
Lance stores it as `FixedSizeBinary(2)`). The gap is bridged with
`bytemuck::cast_vec::<bf16, u16>`, which reinterprets the allocation in
place.

## Why it is sound

- `bf16` is `#[repr(transparent)]` over `u16` in `half` 2.7, so it has
the same size (2) and alignment (2) as `u16`. `cast_vec` therefore hits
its equal-size/equal-align path and reuses the allocation without
realloc; a mismatch would panic, never reach UB.
- `half`'s `bytemuck` feature (enabled here) derives `Zeroable + Pod` on
`bf16`, satisfying `cast_vec`'s trait bounds.
- The crate-root `#[cfg(not(target_endian = "little"))] compile_error!`
added in lance-format#7511 ties the reinterpretation to little-endian hosts,
matching the `FixedSizeBinary(2)` byte order Lance writes elsewhere.
Big-endian builds fail to compile, so the output is byte-identical to
the previous per-element `to_le_bytes()` path.

## Changes

- `From<Vec<bf16>>`: replace the `MutableBuffer` copy loop with
`bytemuck::cast_vec` + `Buffer::from_vec`.
- `half`: add the `bytemuck` feature; add `bytemuck` (`default-features
= false`, `extern_crate_alloc`) as a direct workspace dependency
(already resolved transitively, now promoted).
- Refresh `python/Cargo.lock` and `java/lance-jni/Cargo.lock` (the
workspace-excluded lockfiles) to record `bytemuck_derive`.
- `as_slice`'s alignment doc/SAFETY comment previously asserted every
value buffer comes from `MutableBuffer` (≥32-byte aligned); reworded to
also cover the new `Buffer::from_vec::<u16>` path (2-byte aligned), both
of which satisfy `bf16`'s 2-byte requirement.

## Tests

`test_basics` now pins the raw little-endian bytes emitted by
`From<Vec<bf16>>` (`[0x80,0x3F, 0x00,0x40, 0x40,0x40]` for `[1.0, 2.0,
3.0]`) via `FixedSizeBinaryArray::value`, so a layout or byte-order
regression is caught directly rather than only through Debug formatting.
The prior `assert_eq!(array, array2)` compared two outputs of the same
`From` impl and could not catch a regression.

`cargo fmt`, `cargo clippy -p lance-arrow --tests --benches -- -D
warnings`, and `cargo test -p lance-arrow` (93 unit + 6 doc tests) are
green.

…ance-format#7472)

## Problem
Updating an Arrow JSON column through `fragment.update_columns` / the
update merge path fails with a type-mismatch error.

## Root cause
The HashJoiner right-side stream did not convert Arrow JSON (Utf8) to
Lance JSON (LargeBinary/JSONB) before joining.

## Fix
Wrap the right-side stream in a `JsonConvertingReader` that converts
Arrow JSON fields to Lance JSON before the join.

## Test
Python `test_fragment_update_columns_with_json_column` plus dataset
update tests.

---------

Co-authored-by: xiaojiebao <xiaojiebao@xiaomi.com>
Co-authored-by: Claude Opus 4 <noreply@anthropic.com>

## Summary
- Add "Cleanup old versions" section to the Table Maintenance guide,
explaining versioning storage costs, snapshot isolation, time travel,
the `older_than` parameter, and the `delete_unverified` flag
- Add "Automatic cleanup" section documenting `AutoCleanupConfig`,
`enable_auto_cleanup`/`disable_auto_cleanup`, and dataset config keys
- Add "Other cleanup strategies" section mentioning periodic background
cleanup

## Test plan
- [ ] Verify docs render correctly with `mkdocs serve`
- [ ] Review code examples for accuracy against current Python API

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… of `Error::IO` (lance-format#6569)

Closes lance-format#2067

### Summary

The blanket `From<object_store::Error>` impl mapped **all** object-store
errors to `Error::IO`, losing the semantic meaning of "not found." This
forced downstream code into fragile multi-level downcasting and left
several `Error::NotFound` match arms unreachable for the object-store
path (e.g., in `dataset.rs`, `builder.rs`, `insert.rs`).

This PR:
- Discriminates `object_store::Error::NotFound` in the `From` impl so it
converts to `Error::NotFound`
- Simplifies two call sites that relied on downcasting (`cleanup.rs`,
`refs.rs`)
- Adds tests for the conversion and `#[track_caller]` location
propagation

### Breaking Changes

`object_store::Error::NotFound` now produces `Error::NotFound` instead
of `Error::IO`. Code matching `Error::IO` to detect object-store
not-found conditions (via downcasting) will stop catching them. Match on
`Error::NotFound` instead. This is expected during the current
`6.0.0-beta.1` pre-release cycle.

### Design Notes

- The `source` field from the object-store variant is intentionally
dropped — `Error::NotFound` carries only `uri` and `location`,
consistent with all other `Error::not_found()` call sites in the
codebase.
- Call sites matching raw `object_store::Error::NotFound` before
conversion (e.g., `exists()`, commit resolution, external manifests) are
unaffected.

…#6606)

## Summary
- Adds a **Fragment Sizing** subsection to the Performance Guide that
frames the core tradeoff between manifest-level operations (cost scales
with fragment count) and fragment-level operations (cost scales with
fragment size, drives per-fragment conflict detection).
- Gives practical guidance: 1M rows/fragment default holds up to ~1B
rows, tens of thousands of fragments is generally fine, 10 GB–100 GB is
a reasonable per-fragment upper range (1 TB hard ceiling), and
concurrent update/delete/merge_insert workloads should err toward more
fragments.

## Test plan
- [ ] `mkdocs serve` renders the new section cleanly under Performance
Guide.
- [ ] Cross-link to `../format/table/transaction.md#conflict-resolution`
(already present in the surrounding section) still resolves.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…7616)

## What

`arrow-array` was declared in both `[dependencies]` and
`[dev-dependencies]` of the `fsst` crate. A normal dependency is already
visible to tests, examples, and benches, so the `[dev-dependencies]`
entry was redundant. This drops the duplicate line.

## Tests

No behavior change — purely a dependency-manifest cleanup. `cargo test
-p fsst`, `cargo build -p fsst --example benchmark`, and `cargo clippy
-p fsst --tests --benches -- -D warnings` are green. No lockfile change,
since `arrow-array` remains a workspace dependency.

…ping to TableNotFound (lance-format#7931)

### Description

During a stress run on a popular cloud provider, 503 errors when listing
objects failed and the dir namespace reported the affected tables as
non-existent. Everything downstream was handed a `Table not found: table
id 'foobar'` with no evidence of a throttle.

Another problem is also a correctness issue, any caller that asks `does
this table exist?` could mistakenly be told yes during a throttle burst
and get the wrong answer.

This change returns ThrottleErrors to callers unless there is a genuine
miss in the object store

## What changed

- add native BloomFilter segment merging with fragment filtering and
conservative legacy null handling
- wire BloomFilter segment detection and merge dispatch into dataset
index creation
- enable Python distributed segment creation and uncommitted index
publication for BloomFilter indexes
- add Rust unit/integration coverage and a Python end-to-end test

## Why

Distributed index construction could create segmented scalar indexes for
several index types, but BloomFilter still rejected fragment-scoped
builds and could not consolidate segment metadata into a queryable
index.

## Impact

BloomFilter scalar indexes can now be built per fragment group, queried
as logical segments, merged while excluding retired fragments, and
committed through the Python API.

## Summary

- stop pushing a finite source limit below cross-generation
`PkBlockFilterExec`
- let the existing post-filter `LocalLimitExec` pull until it has
`offset + limit` live rows or reaches EOF
- retain limit pushdown for sources without a block filter
- add regressions for update/filter predicate crossing, active
tombstones, multiple base fragments, and offset/limit semantics

Closes lance-format#7916

## Why

A stale prefix could consume an on-disk source's finite limit before
cross-generation shadow filtering removed it. The scan then returned
fewer rows than requested even though later live rows existed. A finite
over-fetch factor cannot make that exact.

Block-filtered scans now use the existing execution tree as a streaming
refill: `LocalLimitExec` stays above `PkBlockFilterExec`, continues
pulling until enough rows survive, and cancels the child once the source
contribution is full. Sources without a block filter keep the physical
limit pushdown.

## Testing

- `cargo test -p lance --lib test_lsm_scan_limit_`
- `cargo test -p lance --lib dataset::mem_wal::scanner::`
- `cargo test -p lance --lib dataset::mem_wal::`
- `cargo fmt --all`
- `cargo clippy --all --tests --benches -- -D warnings`
- pre-commit hooks: fmt, typos, and Cargo lock sync

## Compatibility

No public API, file format, or dependency changes.

## Classification

This fixes silently incomplete query results and should be labeled
`critical-fix`. The linked issue uses the required `bug:` title prefix,
but GitHub rejected label mutation for the contributor account.

…ance-format#7651)

## Summary

The mini-block structural scheduler caches per-page state that is
consulted on
every read. Today that state is a `Vec<ChunkMeta>` (num_values, size,
offset per
chunk) **plus** a repetition index of per-chunk blocks (first_row,
starts_including_trailer, has_preamble, has_trailer) — ~48 bytes/chunk,
almost
all of it derivable from a handful of cumulative quantities.

This PR replaces both with a compact `MiniBlockChunkIndex` (new
`primitive/chunk_index.rs` submodule) that stores only the non-redundant
data
and derives the rest:

- **`byte_starts: PrefixSums`** — cumulative chunk byte sizes, stored as
`u32`
  when the page's data buffer fits in 4 GiB and `u64` otherwise.
- **`rows: RowMapping`** selected by page shape:
- `UniformFlat` — fixed-width / bitpacked pages where every non-last
chunk
holds the same number of values; row↔chunk is pure arithmetic with **no
    heap allocation**.
- `Flat` — non-uniform flat pages (RLE / FSST); a single cumulative
value
    array.
- `Nested` — repetition present; cumulative row starts + a `has_trailer`
bitmap (preamble derived from the previous chunk's trailer) + leaf item
    counts.

`byte_range`/`items_in_chunk`/`find_chunk`/`first_row`/`rows_in_chunk`/
`has_preamble`/`has_trailer` preserve the exact semantics the scheduler
relied
on (`find_chunk` still returns the first of duplicated start rows).
Row→chunk
lookups remain O(1) (uniform flat) or O(log n) (binary search), and the
multi-range instruction merge in `schedule_instructions` now runs in
place
instead of allocating a second `Vec`.

`MiniBlockChunkIndex` implements `DeepSizeOf` (composing the inner
`PrefixSums`/`ItemCounts`/`RowMapping`), so cache weighing reflects the
smaller
footprint.

This is an idiomatic port of an internal change to current upstream: it
uses
lance's `DeepSizeOf` trait (rather than a bespoke size method), threads
the
existing `initialize` IO structure, and drops fork-only test scaffolding
that
has no counterpart here.

## Test plan

- [x] `cargo test -p lance-encoding --lib` — 422 passed, 0 failed, 5
ignored
(covers flat, non-uniform-flat, and nested/list miniblock round trips,
      e.g. `test_sparse_large_string_list::...MINIBLOCK`,
      `test_sparse_boolean_list_uses_miniblock`, `test_nested_strings`).
- [x] New unit tests: page-shape detection (`test_flat_detection`
cases),
nested row/item axes (`test_nested_detection_and_axes`), uniform-vs-Flat
scheduler/lookup parity (`test_uniform_flat_matches_prefix_sum_flat`),
      per-variant heap-size bounds vs the legacy layout
(`test_deep_size_per_variant_below_legacy`), and a `parse_nested_rep`
      oracle test against the previous per-block decode.
- [x] `cargo fmt -p lance-encoding`
- [x] `cargo clippy -p lance-encoding --tests --benches -- -D warnings`

Made with [Cursor](https://cursor.com)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

… group across 1 directory (lance-format#7920)

Bumps the uv group with 1 update in the /python directory:
[duckdb](https://github.com/duckdb/duckdb-python).

Updates `duckdb` from 1.5.4 to 1.5.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/duckdb/duckdb-python/releases">duckdb's
releases</a>.</em></p>
<blockquote>
<h2>v1.5.5 Bugfix Release</h2>
<p>See <a
href="https://github.com/duckdb/duckdb/releases/tag/v1.5.5">DuckDB's
changelog</a> for all changes in DuckDB.</p>
<h2>What's Changed in DuckDB-Python</h2>
<ul>
<li>Fix numpy deprecations by <a
href="https://github.com/evertlammerts"><code>@​evertlammerts</code></a>
in <a
href="https://redirect.github.com/duckdb/duckdb-python/pull/505">duckdb/duckdb-python#505</a></li>
<li>Fix alias of <code>DuckDBPyRelation.query</code> (closes <a
href="https://redirect.github.com/duckdb/duckdb-python/issues/468">#468</a>)
by <a
href="https://github.com/evertlammerts"><code>@​evertlammerts</code></a>
in <a
href="https://redirect.github.com/duckdb/duckdb-python/pull/524">duckdb/duckdb-python#524</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/b236c8194ed14c7a7c685e0534dde501cc855b3a"><code>b236c81</code></a>
Pin submodule to release hash</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/e6a21c5b5d65d741fafd5a7389c93b8b86a7f249"><code>e6a21c5</code></a>
[duckdb-labs bot] Bump DuckDB submodule (<a
href="https://redirect.github.com/duckdb/duckdb-python/issues/554">#554</a>)</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/f637bf42683030cbb108268581b866f13cb75031"><code>f637bf4</code></a>
Bump submodule</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/4c5c73c63381bd0587fd42b41093c8f0abe9f17c"><code>4c5c73c</code></a>
[duckdb-labs bot] Bump DuckDB submodule (<a
href="https://redirect.github.com/duckdb/duckdb-python/issues/551">#551</a>)</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/6fae28061912de88c3c0f6e6ad6b2f3fc5ee5235"><code>6fae280</code></a>
Bump submodule</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/d8d5f0a36003c9090ea9480cdefa8f4b4122fece"><code>d8d5f0a</code></a>
[duckdb-labs bot] Bump DuckDB submodule (<a
href="https://redirect.github.com/duckdb/duckdb-python/issues/548">#548</a>)</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/5d85f06025e83e3d2c69813acf4b2dc02bebc415"><code>5d85f06</code></a>
Bump submodule</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/c3cbf8d9ca87cea3dfbb89bfa2102ff4ec530f00"><code>c3cbf8d</code></a>
[duckdb-labs bot] Bump DuckDB submodule (<a
href="https://redirect.github.com/duckdb/duckdb-python/issues/545">#545</a>)</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/93418b65c6ca821d71d9a091fb5218ad55d8a4eb"><code>93418b6</code></a>
Bump submodule</li>
<li><a
href="https://github.com/duckdb/duckdb-python/commit/24f1915b590e1e663a14c0bff9e49bf28ebece9b"><code>24f1915</code></a>
[duckdb-labs bot] Bump DuckDB submodule (<a
href="https://redirect.github.com/duckdb/duckdb-python/issues/539">#539</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/duckdb/duckdb-python/compare/v1.5.4...v1.5.5">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=duckdb&package-manager=uv&previous-version=1.5.4&new-version=1.5.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ance-format#7900)

## Summary

Ports the public-surface regression test from the closed lance-format#7759 onto the
post-lance-format#7888 design.

- Add an end-to-end MemWAL test through the public `ShardWriter` API:
after a MemTable rotation, a peer writer claims a higher epoch, and the
first durable put into the new generation must wait for its own WAL
flush and surface the `PeerClaimedEpoch` fence — not ack from the stale
generation's durability state.
- Organize the integration coverage under `rust/lance/tests/mem_wal/`
and add a durable public `put` → `scan` round-trip test for the primary
write/read path.
- This is the integration-level counterpart to the lib-level
`test_durable_ack_after_rotation_requires_its_own_wal_append` added in
lance-format#7888, which covers the false acknowledgement (lance-format#7760) via internal
stats; this test exercises the same failure sequence purely through the
public API (`open` / `put` / `force_seal_active` /
`wait_for_flush_drain`) and asserts the typed `FenceReason`.

## Background

Batch positions restart at 0 in every MemTable generation. Before the
writer-global durability cursor (lance-format#7888), a durability watch keyed on the
generation-local position could be satisfied by the previous
generation's flush at the same position — the first durable put after a
rotation returned success before its own WAL append completed. The
sequence here (rotate → peer claims epoch → durable put at the same
local position) makes that false success observable as a missing
`PeerClaimedEpoch` error.

## Pre-lance-format#7888 failure sequence (what the test must never allow again)

```mermaid
sequenceDiagram
    autonumber
    participant C as Client
    participant G0 as Writer A / generation N
    participant M as Shared durability watermark
    participant G1 as Writer A / generation N+1
    participant B as Writer B
    participant W as WAL

    C->>G0: durable put, local range 0..1
    G0->>W: flush generation N
    W-->>G0: success
    G0->>M: publish watermark 1
    Note over G0,G1: MemTable rotates and local positions restart at 0
    B->>W: claim a higher writer epoch
    C->>G1: durable put, local range 0..1
    G1->>M: wait for watermark >= 1
    M-->>G1: already satisfied by generation N
    G1-->>C: incorrect success
    G1->>W: flush generation N+1
    W-->>G1: PeerClaimedEpoch
```

Steps 1–4 map to the first `put` plus `force_seal_active` +
`wait_for_flush_drain`, step 6 to opening writer B, and steps 7–13 to
the second `put`.

## Post-lance-format#7888 behavior (what the test asserts)

The durability cursor is writer-global and the put's target is lifted
through `BatchStore::global_offset()`, so the wait can only be satisfied
by an append covering this generation's own range — and the peer fence
arrives *before* any success can be reported:

```mermaid
sequenceDiagram
    autonumber
    participant C as Client
    participant G1 as Writer A / generation N+1
    participant Cur as WriterCursors (writer-global)
    participant H as WalFlushHandler
    participant W as WAL

    C->>G1: durable put, local range 0..1
    G1->>G1: durable target = global_offset(1) + 1 = 2
    G1->>H: trigger flush of generation N+1
    G1->>Cur: wait until durable() >= 2 (cursor at 1)
    H->>W: append generation N+1 range
    W-->>H: PeerClaimedEpoch (writer B claimed the epoch)
    H->>Cur: latch terminal error, wake waiters
    Cur-->>C: Error::Fenced(PeerClaimedEpoch)
```

The test asserts exactly step 8: `error.fence_reason() ==
Some(FenceReason::PeerClaimedEpoch)`.

## Validation

- `cargo test -p lance --test integration_tests mem_wal:: --
--nocapture` — passed (2/2)
- `cargo fmt --all -- --check` — passed
- `cargo clippy --all --tests --benches -- -D warnings` — passed
- pre-commit hooks (fmt, typos, lock-sync) — passed

## Compatibility

Test-only change; no API, format, or dependency changes.

)

## Why

Blob selection APIs currently expose an implementation detail of the
physical read planner: null descriptors are omitted because they
schedule no payload I/O. This makes result cardinality depend on
descriptor contents, prevents callers from distinguishing null values
from omitted rows, and leaves `read_blobs`, `read_blob_ranges`, and
`take_blobs` with inconsistent contracts.

This is a follow-up to lance-format#7864 and resolves the remaining null behavior
described in lance-format#7899.

## Contract

All blob selection APIs now return one logical result per selector or
range request. Null blobs are represented explicitly, while valid empty
blobs and empty ranges remain non-null empty values. The physical
planner still avoids payload I/O for nulls; the logical result layer
restores their selection positions.

## Breaking change

This intentionally changes public return types and observable result
cardinality:

- Rust `take_blobs*` APIs return `Vec<Option<BlobFile>>`, and
`ReadBlob::data` / `ReadBlobRange::data` are `Option<Bytes>`.
- Python blob APIs return `Optional` values for null blobs.
- Java `takeBlobs*` lists retain every requested position and may
contain null elements.

Callers that relied on null selections being omitted must now handle
explicit null results.

BREAKING CHANGE: Blob selection APIs now preserve every selector or
request and represent null blob values explicitly instead of omitting
them.

…ce-format#7943)

## What

A flushed MemTable generation always carries a BTree primary-key index
and is a persisted, immutable Lance dataset — i.e. an **SSTable** in LSM
terms. This renames the internal terminology for that unit.

- The persisted-unit noun — `FlushedGeneration` / `flushed_generations`
/ `flushed MemTable` — becomes `SsTable` / `sstables`. "sstable" already
implies flushed, so the qualifier is dropped.
- **Kept unchanged:** the flush *verb*, WAL-durability terms
(`all_flushed_to_wal`, `rows_flushed`, `unflushed_memtable_bytes`), and
the generation *number* concept (`LsmGeneration`, `MergedGeneration`,
`current_generation`, on-disk `_gen_{i}`). An SSTable is *identified by*
its generation number.

## Surfaces

`protos/table.proto` (message `SsTable`, field `sstables` — field
numbers preserved), the `lance-table` core types,
`rust/lance/src/dataset/mem_wal/` (incl. `LsmDataSource::SsTable`,
`SsTableCache`, `open_sstable`, `sstable_cache.rs`), the Python and Java
bindings, the mem_wal benches, and `docs/src/format/table/mem_wal.md`.

## Compatibility

MemWAL is experimental. Proto field numbers are unchanged
(wire-compatible), and `ShardManifest` persists as protobuf, so there is
no on-disk change. Experimental binding APIs are renamed directly
without deprecation shims.

…at#7615)

**merge_insert** directly invokes **session_ctx.read_lance_unordered**
without specifying Blob column handling, which defaults to
**BlobsDescriptions**. Consequently, DataFusion retrieves Blob data as a
**Struct** type description when scanning the original dataset, causing
a crash with "**LargeBinary vs Struct schema mismatch!**" because the
table's logical schema expects LargeBinary.
Interestingly, this only happens if the input data for **merge_insert**
lacks Blob columns; if Blob columns are present in the input, column
pruning is triggered during the original data read, bypassing the Blob
columns and masking the bug.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configurable blob handling for scans via an optional
`blob_handling` setting on the table provider.
* Introduced a builder method to set blob handling when creating a
provider with ordering.
* **Bug Fixes**
* Improved merge-insert/upsert planning/execution for datasets with
nullable blob-typed columns, preserving existing blob values and
correctly nulling missing blobs on inserts.
* **Deprecations**
  * Deprecated the unordered read API in favor of the ordered read API.
* **Tests**
  * Added coverage for merge-insert/upsert with nullable blob columns.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: niuyulin <niuyulin@chinamobile.com>

…null API (lance-format#7963)

## Summary

`take_blobs*` now returns `Vec<Option<BlobFile>>` (after the take_blobs
null API change).

Update the new `merge_insert` blob assertions to match that contract so
the lib tests compile again.

## Test plan

- `cargo test` in `rust/lance` (lib tests) passes.

Adds fragment-scoped RTree index training and native segment
consolidation.

The merge path preserves retained geometry and null rows, applies
fragment-reuse remapping before filtering, and rebuilds a canonical
RTree segment. Python routing and end-to-end segmented build, merge,
commit, and query coverage are included.

Validated with Rust geo checks, RTree tests, clippy, formatting, Python
lint, and the geo integration test.

…-format#7927)

Addresses lance-format#2628.

## Problem

With `prefilter=true`, the HNSW sub-index runs the full beam search and
only gates the result heap by the mask. Every visited node gets a
distance computation whether or not it can be returned, so under
selective filters most of the work is wasted. The worst case is a filter
whose matches cluster far from the query's region of the graph: on
SIFT1M that costs 9-240 ms per query (table below).

## Change

Adds `beam_search_acorn`, a filtered traversal in the style of ACORN-1
([Patel et al. 2024](https://arxiv.org/abs/2403.04871)), used by the
HNSW sub-index for prefiltered queries when the query opts in with
`ApproxMode::Fast`. `Normal` (the default) and `Accurate` keep the
existing traversal, so default behavior is unchanged.

- Distances are computed only for mask-passing nodes. A filtered-out
neighbor contributes its own neighbors instead (expanded at most once).
- Masked nodes two hops out are kept as unscored waypoints. If the
frontier starves before `min(ef, matching)` results, waypoints are
expanded under a budget (`4 * ef`), so passing components behind masked
chains deeper than one node are still reached. This addresses the review
counterexample, whose regression test is included.
- The frontier is seeded with 16 nodes sampled from the mask, so the
search cannot strand when the passing set sits across a masked region
from the entry point.
- Dispatch safety nets: a mask that passes every row shortcuts to the
unfiltered path, filters below 10% of a partition keep the existing
exact flat fallback, and if the traversal still under-delivers (budget
exhausted on a fragmented mask) it falls back to the existing traversal.
Range-bounded queries skip that fallback since they return short
legitimately.

The traversal is generic over the vector storage, so `Fast` covers
IVF_HNSW_FLAT, IVF_HNSW_SQ, and IVF_HNSW_PQ, on both the in-memory and
disk-loaded graph backends. Deletion masks compose unchanged since the
bitset is built from the existing `PreFilter`.

## Benchmarks

SIFT1M (1M x 128, L2), single graph, default build params, k=10, median
of 20 official queries, Apple M-series. Unfiltered control: 0.30 ms at
1.000 recall@10 against the official ground truth. `current` is today's
traversal at ef=100. Cells are latency / recall@10.

Filter matches a cluster unrelated to the query (the pathological case
today):

| selectivity | current | ACORN ef=100 | ACORN ef=400 | flat scan |
|---|---|---|---|---|
| 2% | 37.3 ms / 1.000 | 0.88 / 0.975 | 2.00 / 1.000 | 1.81 / exact |
| 5% | 217.5 / 0.990 | 0.88 / 0.960 | 2.31 / 1.000 | 4.86 / exact |
| 10% | 27.3 / 0.970 | 0.97 / 0.925 | 2.45 / 0.990 | 10.2 / exact |
| 25% | 22.1 / 0.960 | 0.78 / 0.945 | 1.98 / 0.960 | 26.3 / exact |
| 50% | 8.6 / 0.990 | 0.89 / 0.960 | 2.09 / 0.990 | 52.6 / exact |

At the same ef the traversal is 10-250x faster and gives up at most 4
points of recall. Since ef is a per-query parameter, spending some of
that headroom (ef=400) matches the current traversal's recall on every
row while staying 4-100x faster.

Filter matches the query's own region: 0.09-0.27 ms at recall equal to
the current traversal (which costs 0.34-0.60 ms). Uniform random masks:
0.85-1.45 ms at 0.945-1.00 vs 0.84-6.49 ms at ~1.00, converging at 25%+
selectivity.

GIST1M (1M x 960, L2, unfiltered ceiling 0.820 at ef=100) confirms the
story at higher dimensionality, where the flat scan stops being
competitive even at 2% selectivity (10 ms). Clustered filters at 2%: 2.4
ms / 0.985 vs 72.8 ms / 0.985 for the current traversal, and `Fast` at
ef=400 reaches 0.990-1.000 at 5-7.6 ms.

Where it loses, so it is on the record: uniform random masks at 2%
selectivity drop recall (0.775 vs 0.975 on GIST1M), and at 50% random
the waypoint bookkeeping makes it slower than the current traversal
(15.3 vs 4.1 ms). Random masks spread the passing set too thin for seeds
plus bridges. Clustered filters (categories, languages, tenants) are the
win case, and the opt-in leaves that judgment with the caller.

Reproduce with `examples/acorn_bench_sift.rs` (`SIFT_DIR=... cargo run
--release -p lance-index --example acorn_bench_sift`) and
`examples/acorn_bench.rs` (synthetic, no download). Happy to drop or
relocate the examples if you'd rather not carry them.

## Tests

- Review counterexample as a regression test: passing components behind
two-node masked chains are all found
(`test_acorn_reaches_across_masked_chains`).
- `test_acorn_filtered_search`: results are mask-only, built and loaded
graphs return identical results, recall floors at default ef and on
deletion-style (nearly all-pass) masks.
- `test_subindex_prefilter_dispatch`: dense masks in both modes,
all-pass shortcut equals unfiltered, sparse masks take the exact flat
scan.
- `test_ann_prefilter` extended to IVF_HNSW_FLAT / PQ / SQ x
`ApproxMode::Normal` / `Fast` (32 cases).

## Notes

- Follows the review suggestion: opt-in via the existing `ApproxMode`
query setting, no new API surface, default path untouched. If the recall
coverage looks sufficient down the line, making `Fast` the
dense-prefilter default can be revisited separately.
- The VBASE direction linked in lance-format#2628 is a broader query-engine-level
design. This PR is intended as a self-contained increment, not a
replacement for it.

…ance-format#7478)

Previously, shutting down an application with concurrent reads ongoing
caused a panic that looks something like this:
```
 Panic!, thread_name: "tokio-rt-worker", msg: Some("range end out of bounds: 128 <= 0"), file_line: Some("/home/ilya/.local/share/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/bytes-1.11.1/src/bytes.rs:392"), backtrace:   
    ...
   6: bytes::bytes::Bytes::slice
   7: lance_io::scheduler::FileScheduler::submit_request::{{closure}}
   8: <lance_file::io::LanceEncodingsIo as lance_encoding::EncodingsIo>::submit_request::{{closure}}
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
  10: <lance_encoding::encodings::logical::primitive::MiniBlockScheduler as lance_encoding::encodings::logical::primitive::StructuralPageScheduler>::schedule_ranges::{{closure}}
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
  12: <lance_encoding::encodings::logical::primitive::StructuralPrimitiveFieldSchedulingJob as lance_encoding::decoder::StructuralSchedulingJob>::schedule_next::{{closure}}::{{closure}}
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
```

This was caused by `MutableBatch` returning an empty buffer on Drop, if
it did not yet receive any data.

With this fix, it will be an I/O error instead.

…rmat#7965)

## Summary

- document generations as SSTable properties rather than MemTable
properties
- rename MemWAL compaction progress to `CompactedSsTable` /
`compacted_sstables` across protobuf, Rust, Python, and Java
- use SSTable Compaction terminology and replace the MemWAL overview and
shard diagrams

## Validation

- `cargo clippy --all --tests --benches -- -D warnings`
- `cargo test -p lance compacted_sstables`
- Python build, lint, and MemWAL tests (20 passed)
- Java and JNI tests (424 passed, 26 skipped) and clippy
- `uv run mkdocs build`

## Why

Ubuntu 24.04 currently provides QEMU 8.2.2 to the pre-Haswell CI job.
That version can crash inside x86_64 user-mode while reading the
vsyscall mapping ([QEMU
lance-format#2170](https://gitlab.com/qemu-project/qemu/-/issues/2170)), so the job
exits with a host-side SIGSEGV before it can detect guest SIGILL
regressions.

Pin and checksum QEMU 8.2.10 from the official release archive, build
only the `x86_64-linux-user` target, cache the resulting binary by QEMU
version and runner platform, and retain `-cpu Nehalem`. This makes the
check exercise Lance runtime SIMD dispatch instead of an old emulator
bug.

Closes lance-format#7902.

---------

Co-authored-by: Lu Qiu <luqiujob@gmail.com>

## Problem

External manifest finalization reused staging object metadata for
manifests smaller than 5 MiB. Object-store copies may assign different
metadata to the destination object, so finalized-manifest validation can
reject a valid table as corrupt when the stored ETag does not match the
copied object.

## Fix

Always read the finalized object metadata after copy and persist its
destination size and ETag. This applies to both the default
external-store commit path and recovery finalization, so the external
store and returned manifest location describe the same object.

Lance FTS execution plans currently distinguish only between all
committed segments and an already-resolved exact metadata subset. A
distributed caller that has an exact ordered UUID scope but cannot await
metadata during plan construction must otherwise resolve metadata early
or risk broadening execution to every committed segment.

This PR adds a native exact-but-unresolved selection state for
`MatchQueryExec` and `PhraseQueryExec`. UUIDs are bound against the
exec's existing dataset snapshot when the output stream is polled,
preserve caller order and duplicates, and fail closed for empty,
missing, or inconsistent metadata. Existing all-committed and
pre-resolved paths continue through the same prefilter, scorer, and BM25
pipeline, with binding time exposed as `fts_segment_bind_duration`.

The change is confined to execution-plan runtime state and does not
alter dataset, manifest, index, or wire formats. It intentionally avoids
plan-lifetime caching so repeated execution retains the existing
metadata-resolution semantics.

## Problem

`InvertedIndex::bm25_search` spawns one cpu-pool task per partition
(~350 per query on a 100-shard index). Under concurrent load this has
two costs:

1. **Dispatch flood**: at a few hundred qps this is ~100K tiny task
dispatches/sec through the blocking pool.
2. **Locality loss**: tens of thousands of in-flight tiny tasks
interleave across cores, so the scoring loops (WAND/MAXSCORE, posting
decompression) run with cold caches. Measured on a 320-core node: the
same per-query scoring work costs ~5x more CPU at c64 than the batched
layout, with identical `index_comparisons` (ANALYZE-verified — the
comparison *count* is unchanged; the cost per comparison is what
inflates).

## Change

Partitions are searched in chunks of `LANCE_FTS_SEARCH_CHUNK` (default
16):

- each chunk loads its postings and scoring DocSets **concurrently**
(async, cache-backed),
- then **one** cpu-pool task scores the chunk's partitions back-to-back
on one thread,
- chunks pipeline against each other via `buffer_unordered`, so chunk N
scores while chunk N+1 loads.

A secondary benefit: partitions scored consecutively on one thread
observe each other's published shared top-k floor immediately.

`LANCE_FTS_SEARCH_CHUNK=1` restores the old one-task-per-partition
behavior.

## Results

320-core node, 100M-doc 42-language corpus, 5-term `match_any` (OR),
tier-100-200 words, k=100, warm + prewarmed, exact scoring, 180s
windows, bench returns `_rowid` + `_score` only:

| concurrency | per-partition tasks (before) | chunked (after) |
|---|---|---|
| 16 | 227 qps / 71 ms | **428 qps / 37 ms** |
| 64 | 224 qps / 286 ms / 78% CPU | 221 qps / 290 ms / **29% CPU** |

At c64 throughput is capped by a separate cache-read bottleneck (a
follow-up PR swaps the backend); the chunked layout cuts the CPU burned
per query ~5x (1.11 → 0.20 core-seconds), which that follow-up then
converts into throughput: with the contention-free read path the chunked
layout reaches **1340 qps at c128 vs 345 for per-partition tasks**.

## Tests

`cargo test -p lance-index scalar::inverted` — 332 passed. The
multi-partition tests (41 partitions = 3 chunks) exercise the
multi-chunk merge path with exact row_id assertions.

…e-format#7988)

Dependabot security updates run independently of the version-update
config in `.github/dependabot.yml`: they ignore the weekly schedule,
were not grouped (groups default to `version-updates`), and scan every
manifest — including ecosystems (`maven` in `/java`, `pip` in
`/benchmarks/*`) and directories (`/memtest`, `/test_data/.../datagen`)
the config never listed. The result was a stream of individual
security-fix PRs (e.g. the same quinn-proto advisory opening four
separate PRs).

This groups them:

- Add an `applies-to: security-updates` group to every ecosystem so
advisories that land together in a directory batch into a single PR
instead of one-per-advisory.
- Bring the previously unmanaged surfaces (`maven` `/java`, `pip`
`/benchmarks/*`, and the standalone cargo lockfiles `/memtest` and
`/test_data/fri_straddle_pre_6610/datagen`) under config with
`open-pull-requests-limit: 0`, so their security fixes are grouped but
no new weekly version bumps are opened.
- Consolidate the three cargo blocks into one `directories` entry
(behavior-neutral for version updates — still one grouped PR per
directory).

**Known limitations** (GitHub behavior, not config bugs):

- The same CVE across multiple lockfiles stays multiple PRs —
cross-directory single-PR grouping (`group-by: dependency-name`) is
version-updates only.
- Test-fixture lockfiles can't be excluded from security scanning via
config (`exclude-paths` is version-updates only).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…mat#7989)

The 9.0.0 crates.io publish failed while packaging `lance-index-core`:

```
error: readme `README.md` does not appear to exist (relative to `.../rust/lance-index-core`).
```

The crate was added in lance-format#7713 with `readme = "README.md"` but no such
file. `cargo publish --workspace` packages every crate before uploading
any, so the whole release aborted and no 9.0.0 crates reached crates.io.

Two changes:

- Add `rust/lance-index-core/README.md`, matching the sibling internal
crates.
- Add a `cargo package --workspace --no-verify` job to the Rust
workflow. Nothing in CI packaged the crates, so manifest problems that
only surface at publish time (missing readme/license files, path
dependencies without a version) were invisible until the release ran.
`--no-verify` skips rebuilding each packaged crate, which the other jobs
already cover, so the job takes about a minute.

Verified locally: all 25 publishable crates now package cleanly.

Fixes lance-format#7986

…jni (lance-format#7982)

Bumps [quinn-proto](https://github.com/quinn-rs/quinn) from 0.11.14 to
0.11.16.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/quinn-rs/quinn/releases">quinn-proto's
releases</a>.</em></p>
<blockquote>
<h2>quinn-proto-0.11.16</h2>
<h2>What's Changed</h2>
<ul>
<li>0.11.x: upgrade dependencies by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/quinn-rs/quinn/pull/2707">quinn-rs/quinn#2707</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/quinn-rs/quinn/commit/a96949f6cd257c665f544626af4e8ce668a40b30"><code>a96949f</code></a>
Take semver-compatible update for anyhow</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/5429f60d0ee9971770e60f0407d992d3b912d274"><code>5429f60</code></a>
udp: bump version to 0.5.15</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/262a493629acdc979070cd36d801464a2b8dd3e2"><code>262a493</code></a>
proto: bump version to 0.11.16</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/c19b63a04c6eff60684a845fce29fee6d74b1acd"><code>c19b63a</code></a>
Upgrade rustls-platform-verifier to 0.7</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/aff3652c43be3491908ef553fac16610c9ad3e6a"><code>aff3652</code></a>
Disable default features for fastbloom</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/01b2eee2c68b1d73ad09485b440fbfa8f6d3e290"><code>01b2eee</code></a>
Upgrade fastbloom to 0.17</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/2c82013a8cd502f3dd0bba0a4c38a51e564d29cc"><code>2c82013</code></a>
Switch BBR RNG to PCG</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/544dd9ebabf18639cb2041f849ea406100dd3d6d"><code>544dd9e</code></a>
Upgrade to rand 0.10.1</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/a7499b8439e393a6299330111d9c8564cd96c464"><code>a7499b8</code></a>
Bump versions for release</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/7c1970f19b24280af86b1a0a1c2d06d59fc453f0"><code>7c1970f</code></a>
proto: yield error on too many gaps in assembler</li>
<li>Additional commits viewable in <a
href="https://github.com/quinn-rs/quinn/compare/quinn-proto-0.11.14...quinn-proto-0.11.16">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=quinn-proto&package-manager=cargo&previous-version=0.11.14&new-version=0.11.16)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/lance-format/lance/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ri_straddle_pre_6610/datagen (lance-format#7984)

Bumps [quinn-proto](https://github.com/quinn-rs/quinn) from 0.11.14 to
0.11.16.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/quinn-rs/quinn/releases">quinn-proto's
releases</a>.</em></p>
<blockquote>
<h2>quinn-proto-0.11.16</h2>
<h2>What's Changed</h2>
<ul>
<li>0.11.x: upgrade dependencies by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/quinn-rs/quinn/pull/2707">quinn-rs/quinn#2707</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/quinn-rs/quinn/commit/a96949f6cd257c665f544626af4e8ce668a40b30"><code>a96949f</code></a>
Take semver-compatible update for anyhow</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/5429f60d0ee9971770e60f0407d992d3b912d274"><code>5429f60</code></a>
udp: bump version to 0.5.15</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/262a493629acdc979070cd36d801464a2b8dd3e2"><code>262a493</code></a>
proto: bump version to 0.11.16</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/c19b63a04c6eff60684a845fce29fee6d74b1acd"><code>c19b63a</code></a>
Upgrade rustls-platform-verifier to 0.7</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/aff3652c43be3491908ef553fac16610c9ad3e6a"><code>aff3652</code></a>
Disable default features for fastbloom</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/01b2eee2c68b1d73ad09485b440fbfa8f6d3e290"><code>01b2eee</code></a>
Upgrade fastbloom to 0.17</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/2c82013a8cd502f3dd0bba0a4c38a51e564d29cc"><code>2c82013</code></a>
Switch BBR RNG to PCG</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/544dd9ebabf18639cb2041f849ea406100dd3d6d"><code>544dd9e</code></a>
Upgrade to rand 0.10.1</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/a7499b8439e393a6299330111d9c8564cd96c464"><code>a7499b8</code></a>
Bump versions for release</li>
<li><a
href="https://github.com/quinn-rs/quinn/commit/7c1970f19b24280af86b1a0a1c2d06d59fc453f0"><code>7c1970f</code></a>
proto: yield error on too many gaps in assembler</li>
<li>Additional commits viewable in <a
href="https://github.com/quinn-rs/quinn/compare/quinn-proto-0.11.14...quinn-proto-0.11.16">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=quinn-proto&package-manager=cargo&previous-version=0.11.14&new-version=0.11.16)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/lance-format/lance/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ormat#7992)

Extend batch vector search (lance-format#6821) to the indexed/ANN path so a single
multi-query request reads each IVF partition's storage once and scores every
query that probes it, instead of re-running a full single-query plan per
vector and unioning the results (which re-opens the index and rebuilds the
prefilter for each query).

- Add `VectorIndex::search_partitions_batch` + `supports_batch_partition_search`
  (defaulted so non-IVF indices stay explicitly unsupported).
- Implement them for `IVFIndex` with a flat-style sub-index
  (IVF_FLAT/PQ/SQ/RQ): load each distinct partition once and accumulate one
  top-k heap per query, sharing the prefilter across the whole batch.
- Add `ANNIvfBatchExec`, which ranks every query against the centroids, runs
  the shared-scan batch search, merges per-query top-k across deltas, and emits
  `query_index`-tagged results; route to it from
  `Scanner::batch_indexed_vector_search` when the gate below holds.
- Normalize each query vector independently for cosine
  (`normalize_batch_query_for_index`): normalizing the concatenated batch key
  with one global norm would scale each vector by a batch-composition-dependent
  factor and break equivalence with single-query search.

The shared-scan fast path is gated to cases that are provably equivalent to
repeated single-query search: fixed nprobes (`minimum_nprobes ==
maximum_nprobes`), no refine step, an IVF flat-style index, and fully-indexed
fragments. With adaptive nprobes the single-query path applies an
`early_pruning` floor and late-search expansion that the batch path does not,
so those queries fall back to the per-query loop, which stays exact. HNSW,
refine, and mixed indexed/unindexed scans also fall back.

Tests: plan shape; exact batch-vs-repeated-single equivalence (nprobes pinned);
cosine regression; shared prefilter; multi-delta cross-delta merge; and
fallbacks for refine and adaptive nprobes. Python parametrized over L2 +
cosine; a batch-vs-repeated-single ANN benchmark.

Closes lance-format#6822

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>